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s A Multi-Agent System for Automated imag Analysis and 
Understanding 

Background of the Invention 

The present invention relates to the field of image analysis and more specifically to 
methods for automated image segmentation, analysis and understanding by means of a 
1 0 novel Multi-Agent system. 

There currently exists a broad spectrum of image analysis technologies and methods that 
try to provide automated image analysis. However, major problems arise from these 
methods as they are most commonly based on simple spectral thresholding techniques or 
variations thereof. There have been many attempts at providing efficient automated 

15 image analysis methods, some of which have been more or less successful, such as 
multiscale analysis and deformable models. In recent attempts, there have been many 
developments in methods that combine already existing techniques. However, the main 
problem with the combination of existing methods consists in determining the optimal set 
of parameters and order in which the methods should be applied. With the recently added 

20 requirement of symbolic image interpretation in the industry as well as in academia, in 
addition to image segmentation, standard image analysis methods have even more shown 
to be inadequate. To efficiently address these issues, a new theoretical field known as 
Multi-Agent systems has been adopted and formalized by the inventors specifically for 
the problems at hand. Although die basic concepts in Multi-Agent systems are known in 

25 the art, the present invention brings forth completely novel methods therein for allowing 
robust and fully automated image segmentation, analysis and understanding. 

Patent 5,978,520 provides a method and system for interactively generating vector data, 
more specifically to draw vector lines, from drawing data by means of a basic multi-agent 
system. The provided multi-agent system is based on the concept of a supervisor agent 
30 that enforces actions upon the agents it controls when problems or contradiction arise. In 
contrast to the invention described in 5,978,520, the present invention provides an 
automated system and methods for the automatic extraction and quantification of objects 
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5 in images as well as symbolic image interpretation by means of a multi-agent system 
composed of fully autonomous agents. 

Summary of the Invention 

An object of the present invention is to provide an image analysis and segmentation, 
method and apparatus thereof for performing automated image analysis in various fields 
10 of applications where extraction and quantification of objects in images is required and 
where precise and robust analyses are of primary importance. 

Another object of the present invention is to provide a multi-agent system that integrates 
novel concepts in order to achieve full automation and maximum precision in image 
analysis contexts where complex image structures are required to be analyzed. The herein 
15 described invention therefore provides a multi-agent system that integrates the concepts 
of midtiscale-based agent cooperation and sensing, sub-primitive agent specializations, 
dynamic agent parameterization as well as incremental feedback 

Overall there is provided methods and apparatus for first inputting digital images, then 
automatically analyzing the digital images by means of a specialized and autonomous 
20 multi-agent system that is initialized and assigned to the considered images by a program, 
and finally for displaying to the user and for saving in repositories the results generated 
by the multi-agent system. 

Brief Description of the Drawings 

The invention will be described in conjunction with certain drawings which are for the 
25 purpose of illustrating the preferred and alternate embodiments of the invention only, and 
not for the purpose of limiting the same, and wherein: 

Figure 1 displays the basic agent initialization and assignment process. 

Figure 2 depicts the various agent components for an embodiment 

Figure 3 displays the basic flow of an example agent initialization and assignment 
30 process for one embodiment 



2 



5 Figure 4 displays the entire system's basic operational flow. 

Figure S depicts the concept of an agent sensing an image by means of its sensors. 

Figure 6 depicts the basic steps involved in the agent-to-agent communication process by 
means of the Receiver/Transmitter component 

Figure 7 depicts the concept of multiscale agent specialization and cooperation. 

10 Figure 8 is a representation of the components involved in providing multiscale sensors. 

Figures 9a - 9b represent the concept of multiresolution image sensing. 

Figures 10a - 10c depict the process of multiscale information gathering by an agent. 

Figure 11 is a drawing of a typical satellite image landscape. 

Figure 12a - 12b illustrates the extraction of water course ridges and the process of 
1 5 agent initialization on these ridges. 

Figure 13 illustrates the initialization of Border Agents and the path following by Ridge 
Agents. 

Figure 14a - 14b illustrate the bridge detection process. 

Figure 15a - 15b represents the semantic interpretation of water courses. 

20 Figure 16a - 16b illustrate the detection of entities within water courses. 

Figure 17 illustrates a basic configuration of an apparatus for analyzing images 
comprising N processing nodes. 

Referring numerals comprised in the figures are here forth mentioned in the detailed 
description within brackets such as: (2). 

25 Detailed Description 

Although it is not the objective of the present invention to extensively describe hardware 
infrastructures that may be more optimal than standard personal computers as support to 



5 the herein described multi-agent system, we however describe a distributed hardware 
configuration that may be used in the present context 

Distributed Systems 

A multi-agent system is inherently highly distributable; from the fact that individual or a 
group of agents can reside on separate machines and communicate with one another over 

10 a network connection. From this, various types of distributed architectures can be 
considered in the implementation of a multi-agent system. Considering applications such 
as high resolution satellite image analysis, where considerable processing power is 
required to process the data in reasonable time frame, a distributed hardware platform 
may be required. A typical configuration is that of two or more networked computers 

15 where each computer (node) processes information using a program, and where a 
program communicates with another program residing on another computer through a 
communication interface. Patent 6,350,239 discloses a basic distributed software 
architecture for medical diagnostic systems, where distinct modules that perform well 
defined and separate functions are executed on separate computers. The invention 

20 disclosed in 6,350,239 does not however address the challenging problem of distributing 
a single "module" over a plurality of nodes. Contrastingly, the herein described invention 
provides means for distributing a single "module" over a plurality of computers or 
processing nodes, where each node may be host to one or a plurality of software agents 
that each perform a subtask of the global task (module). Through communication and 

25 scheduling of events, the plurality of software agents will achieve the required end-result. 
An example multi-node apparatus is illustrated in Figure 17. Depending on the nature of 
the application, the distributed system can be either loosely or tightly coupled. In the 
former case, the system can be a cluster-based backbone, where each processing node is 
physically located near one another, in a rack mounted configuration for instance, and 

30 where each node is networked to a neighboring node using a high-speed network 
interface and cable (such as 1Gb fiber-optics interconnects). The use of high speed 
networking reduces the communication latency between any program and module 
required to exchange information. In the loosely coupled scenario, the backbone may 
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5 comprise networked computers physically remote to one another between which 
information can be exchanged using a networking protocol such as TCP/IP. 

Although the present invention is detailed with an embodiment using a single personal 
computer as computational backbone, it is understood that it can be extended to a more 
sophisticated architecture such as a cluster system. 

10 Entire System 

The entire system comprises two main segments: A first segment that integrates basic 
hardware and software components that form the basic backbone on which the multi- 
agent system relates, and a second segment that is the multi-agent system itself which 
comprises a plurality of software agents and basic software components. 

15 Referring to Figure 1, the first process involved in the system consists in inputting one or 
a plurality of images by means of an input device (2). The latter can be a standard 
input/output device for reading digital data from a repository, or from an imaging device, 
and can comprise a communication interface to read the data from remote repositories 
over a network such as the Internet Once the image(s) read by the device and stored in 

20 memory, the Multi-Agent initialization and assignment program is instantiated. The 

i 

program instantiates one or a plurality of software agents and makes these software agent 
instances accessible in memory (4). Once the agents are instantiated, the program assigns 
the agents to the images that are required to be analyzed (8). In one embodiment, once the 
agents have been instantiated, they become entirely autonomous entities, making their 

25 own decisions and acting according to their goals and states (6). In one embodiment, the 
agents can be randomly assigned to the images, wherein the program positions each of 
the agents at a random coordinate within the image. In another embodiment, the agents 
are strategically positioned within the image according to a priori knowledge on the 
problem at hand and general image structures, as well as according to the agent's 

30 specialization. In an example embodiment, the initialization and assignment program 
invokes a pre-processing method for detecting local minima within the image as initial 
landmarks for positioning the agents. Referring to Figure 3, this initialization process 
consists in first detecting the minima within the image using the said pre-processing 



5 



5 method (24). For each of the said detected minima, spatial coordinates are extracted by 
means of the said method, wherein the said coordinates serve as the initial positioning 
coordinates, and are saved in memory (26). The initiation and assignment program is then 
input the set of coordinates, for which it automatically assigns the initial agents by 
instantiating each of the agents and registering the latter with their according pixel 
10 coordinate and value (28). From this point, the assigned agents will be autonomous and 
can move within the image to inspect other image regions and associated image pixels. 

Referring to Figure 4, the entire system's overall operational flow comprises four main 
steps: first inputting a digital image to the system (30), then initiating the Multi-Agent 
system (32), followed by the actual step of image analysis by means of the Multi-Agent 
15 system (34), and finally, displaying the segmented image and associated results (36). 

Multi-Agent System 

The current invention relates to a novel multi-agent system devised for the automated 
analysis and segmentation of digital images. 

Agent Architecture 

20 By definition a multi-agent system comprises a plurality of agents. The present multi- 
agent system comprises a plurality of complex agents that have the means to receive 
information, take decisions, communicate, and take action, in an autonomous manner. To 
achieve this, a specific agent architecture that comprises a plurality of components has 
been devised. In one embodiment the components are the following: 

25 - Sensor(s) 

- Processors) 

- Actuators) 

- Receiver/Transmitter 

- Neural Processors) 
30 - State machine (s) 

- Memory unit(s) 

These components provide all the necessary means for creating autonomous and 
minimally 'Intelligent" agents. Each of these agents operates by first receiving stimuli 
from the external environment Referring to Figure 2, an agent first receives input from 
35 either its sensor component (10) or its receiver/transmitter component (16). The sensors 
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5 provide the agent with means for sensing its physical environment, which is in the 
present invention one or a plurality of digital images. In this sense, the sensors provide 
the agent with a view of the digital image, wherein the said view is composed of 
processed or unprocessed pixel values that are within the range of the agent's sensors 
within the image. Referring to Figure 5, in one embodiment the agent has two circular 

10 sensors (38) that are positioned according to the agent's position within the image, as 
depicted by (40) which refers to an image region. Image objects are indicated by (42). 
When the agent senses the image using its sensors, it has an appreciation of the image 
content based on the pixel values, from 0 to 255 for instance, comprised within its 
sensors. Again in Figure 5, the agent's sensors overlay portions of image objects that may 

15 have different pixel values (44 and 46). At this point, the agent receives image 
information from its sensors that indicates it is on image objects. 

As opposed to receiving image information by means of sensors, the agent's 
Receiver/Transmitter component provides means for the agent to communicate with other 
agents and therefore receive and/or transmit valuable information. This communication 

20 can be for instance with objective of negotiating, collaborating, or perhaps simply update 
some information or knowledge cumulated by the entire multi-agent system. In one 
embodiment, when an agent initiates communication with another agent, the said agent 
(48) first queries an Agent Registry (50) that holds a list of agents that are instantiated in 
the system along with their associated key or ID (52). The agent then uses the key or ID 

25 to open a communication channel with the other agent (54). Finally, once the 
communication channel with the other agent is established, the agents are free to 
communicate (56). In another embodiment, the Receiver/Transmitter component is used 
to send information to or receive from a shared memory, where information can be 
gathered and accessed by all agents. This shared memory can be a database or a specific 

30 data structure residing in memory and made available to all agents. 

Once external stimuli are input by the agent, the following steps involve multiple 
components and require the agent to take action, and may also require the agent to make 
a decision before taking action. Referring to Figure 2, the information input by the 
sensors (10) is channeled to both or either the Neural Processor (18) and Processor (12) 
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5 components. On one hand, the Processor performs various type of processing operations 
on the input information, such as, without limitation, edge detection, contrast 
enhancement or computation of statistics. On the other hand, the Neural Processor 
component can perfonn classification tasks on the raw input information or on 
information processed by the Processor. In one embodiment, the Neural Processor 

10 identifies whether the input information is the edge of an object in the image. In general, 
the Neural Processor can identify the information input to it by means of classification or 
can incrementally generate a new knowledge base in conjunction with the Memory Unit 
(22). This incremental aspect of the Neural Processor provides the agent with means for 
learning on its own and can therefore adapt itself to new problems and new types of 

15 images. This can be achieved by first inputting the sensor information in the Neural 
Processor, then automatically classifying the input information and saving a descriptor of 
this information along with its associated class in the Memory Unit. In one embodiment, 
the Neural Processor is comprised of one or a plurality of Artificial Neural Networks, in 
which case the learning process further adapts the Networks* weights. In another 

20 embodiment, the Neural Processor comprises Self Organizing Networks such as Self 
Organizing Maps, Growing Cell Structures, or any variants thereof. The self organizing 
networks can provide powerful means for automatically classifying and characterizing 
the visual information input by the agent's sensors. 

An agent further comprises an Actuator component (14) and can also comprise a State 
25 Machine (20). On one hand, the Actuator is the component that controls the agent's 
process of taking action. It is a general component and can control the process of 
communicating, proper information dispatching, adapting sensors, or even the process of 
moving in the image and changing its content Specific behavior or decision making can 
be achieved in conjunction with the State Machine which serves as the Agent's pre- 
30 defined mental states. In one embodiment, the agent's Actuator will force the adaptation 
of the Agent's sensors based on the output of the Processor, State Machine, Neural 
Processor, or any combination thereof. The sensor adaptation can comprise, without 
limitation, changing the size of the sensors so as to view more or less information at one 
instant within the image. Another adaptation can be to enforce die sensors to view the 
35 information at a specific scale within a multiscale view of the image. In another 



5 embodiment, the Agent will move in a specific direction in the image according to the 
Actuator's command based on the knowledge acquired by the other components. This 
process of moving an agent in the image can take the form: move to coordinate (x,y) and 
sense image using sensors at these new coordinates. In which case, the Agent 
autonomously moves in the image. As the agents* general objectives are to analyze" and 

10 segment images, the Actuator component is also responsible for the action of changing 
pixels within an image, as part of the image segmentation process. When an agent is in 
the mental state of "segmentation", as defined by its State Machine component (see 
below), the agent changes the value of the pixel on which it finds itself. The change in 
value can be of binary form (change any pixel to a 1 or a 0), or can also be of any other 

15 value. Each agent specialization may change the pixel values to their own predefined 
value, which results in associating these specific pixels to their own class. For instance, a 
water agent can change the values of pixels it identifies as "water" to a unique value of 
255, where pixels segmented by other specialized agents will have values different than 
255. In the present invention, the original input images are never directly modified, but 

20 instead copies of the images are created and used by the system. In such case, the 
segmented image that comprises modified pixel values will be done on a copied image, 
used by every agent in the system. 

The agent's State Machine component provides means for preprogramming desired 
behaviors and patterns upon the activation of a specific state. For any given state the 

25 agent is in, if this particular state is coded in the State Machine component, then an action 
or a sequence of actions that is to be taken is directly determined from this component 
Upon definition of an action or set of actions, the State Machine will work in conjunction 
with the Actuator for the agent to execute the required actions. In specific embodiments 
the State Machine component is defined by Decision Trees and Finite State Machines. An 

30 example sub-sequence from the State Machine component could be as follows: 

I- 

| IF sensor input information not classified by Neural Processor 
| Adapt sensors 
| Revalidate input 

35 |... 



The sub-sequence above provides a simple view of how the component can be involved 
in the execution of actions when the agent is in a specific state. In this particular example, 
the agent's Neural Processor was not able to identify the nature of the information input 
by the sensors and therefore finds itself in an "Information not Classified" mental state. 
This state is encoded by the State Machine and defines the various actions required to be 
taken by the agent when such a state is encountered. The example sub-sequence is with 
respect to the dynamic adaptation of the sensors, where when the information is not 
classified, the agent is required to change its sensors' configuration (adapt) and revalidate 
the information input by the later. In one embodiment, when the agent cannot readily 
identify the information as being part of an image object's edge, the sensors dynamically 
change the scale and resolution at which the information is considered, a method that 
may facilitate the identification process. Although the Example State Machine sub- 
sequence is of simple nature, it is understood that the complete decision tree or finite state 
machine may comprise a broad range of states and decision processes. Although not 
directly illustrated in Figure 2, the Receiver/Transmitter component may also be 
controlled in conjunction with the State Machine, where for instance a simple incoming 
communication may trigger a predefined sequence of actions as defined by the 
component 

According to the herein described Agent Architecture, the present Multi-Agent system 
can be composed of specialized agents that make decisions and take actions specifically 
according to their specialization and in accordance with the inherent collaborative aspect 
of the system. From the previous description of the basic components that compose an 
agent, it is clear that an agent can integrate derived components (same components with 
modifications) as well as a plurality of other components. In one embodiment, each agent 
specialization has sensors that can differ in shape, sensitivity, specificity and behavior. 
For instance, an agent that is specialized for the detection of object contours can have 
Neural Processor and Sensors that are specialized for the detection of edges. In another 
embodiment, an agent can be specialized for the analysis of high noise images, where the 
agent will be adapted to perceive the images as a multiscale representation so as to be 
robust to the noise and artifacts. A plurality of agent specializations can be readily 
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5 devised with the herein described agent architecture. It is therefore understood that the 
Invention is not limited to the above described embodiments of agent specialization. 



Multiscale Perception 

Based on the beforehand described agent architecture, an aspect of the present system is 
the multiscale-based agent cooperation and image perception. The concept of multiscale 
10 perception herein considered comprises two main aspects: 1) multiscale perception 
achieved through the cooperation of agents specialized for distinct or multiple scale 
levels, and 2) multiscale perception achieved by means of multiscale and multiresolution 
agent sensors. In the first case, the general concept of scale can be applied to both 
semantic and spatial scales, wherein the second case only applies to spatial scale. 

15 Cooperative Multiscale Perception 

In this first case, a number of agents are specialized and adapted to analyze the image 
represented at certain scale levels, where for instance agents specialized for analyzing an 
image at level 'N* (58) cooperate with one another and further cooperate with agents 
specialized for a level 'N-l* (60). In this context, agents analyzing an image at a scale 

20 level N can facilitate the analysis process by providing new information to agents at finer 
scale analyzing the base image data. As the agents at higher scale have a different and 
broader understanding on the nature of the object or region currently being analyzed, the 
provided information to the lower level agents will allow the latter to adapt their 
parameters and reorient their paths if required. The higher level agents can also enforce 

25 restrictions on the lower level agents, where the agents can be bound to a specific region 
for instance. This multiscale cooperation can also comprise mutual feedback mechanisms 
where the knowledge and information gathered by agents at higher level can facilitate the 
lower level analysis, and where the lower level information can improve and facilitate die 
higher scale analysis. This coevolution provides robust image analysis means as it 

30 simultaneously takes into account multiple scale representations of an image, allowing 
for increased robustness to noise and providing a global and local view of the 
information. Figure 7 depicts this concept of multiscale cooperation where agents on 
multiple scales cooperate with one another in order to efficiently analyze the raw image 
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5 data. The herein described method of cooperative multiscale perception and image 
analysis can further comprise the processes of agent reproduction and agent cloning 
where for instance when an agent that analyzes the raw image data finds itself in an 
uncertain environment or is unsure on the action it should perform due to the degenerated 
information, the agent has the possibility to clone or reproduce itself at higher scales, or 
10 inversely from higher scale to lower scale. This requires the agent to first identify the 
scales at which it will instantiate new agents, then clone or reproduce itself to these 
scales, allow the new agents to analyze the image they find themselves on, and update 
itself according to the information and knowledge provided by the new agents. 

Multiscale Sensor Perception 

15 In the case of sensor based multiscale perception, where only spatial scale is considered, 
the agents are composed of sensors adapted to provide a multiscale representation of the 
portion of the image being sensed. The agent therefore has the possibility of considering 
the image at one spatial scale level or at a plurality of levels within the multiscale 
representation. Referring to Figure 8, this mechanism comprises the steps of first reading 

20 the data that is in range of the sensors by means of the sensors(62), following this, the 
data is then input to the Processor component which computes a multiscale representation 
of the said data (64) using a method known in the art. The multiscale data representation 
can be input to the Neural Processor or further used by the Processor component. In one 
embodiment, the agent considers only one specific scale level of the multiscale image 

25 representation. In another embodiment, the agent sequentially or simultaneously 
considers a plurality of scale levels. In yet another embodiment, the agent perceives the 
image at various spatial scales as well as in multiple resolutions, where the agent's 
sensors change in size to perceive a different level of information. This type of adaptation 
is useful when an agent is required to further validate the environment in which it is 

30 found so as to increase or decrease its confidence of the structure it analyzes. Referring to 
Figure 9.a, the sensor data is input to either or both the Processor and the Neural 
Processor (66) where the data can be analyzed and processed, after which a response is 
sent to the Actuator that changes the size of the sensors (68) according to the later 
response. This process is repeated as required. This sensor adaptation is represented in 
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Figure 9.b, where the initial sensor state is depicted by (70), and where after adaptation 
the sensors are seen to be larger in size (72), meaning that more image data is being read 
by the sensors. In one embodiment, this type of adaptation is used to perceive a larger 
image region to ensure the agent is precisely detecting a contour* 

Agent Negotiation 

In order to resolve problems or contradictions as well as to facilitate the emergence of 
optimal solutions in the image segmentation process, the present system provides a 
means for Agent Negotiation. In a preferred embodiment, the agent negotiation is done 
by agents of different specialization that attempt to analyze the same image region. The 
objective is to allocate the image region to the agent that has maximum confidence on the 
segmented region. The process consists in first identifying the litigious region and for 
each agent to thereafter 

- Evaluate considered region according to specialization; 
• Communicate with other agents in litigious state; 

- Vote for winning agent. 

The first step of identifying the litigious region is achieved in one embodiment by a 
program that verifies if there is a plurality of agents that attempt to analyze a same image 
region. In another embodiment, an agent itself verifies if any other agents are analyzing 
the current region. In such case, the current agent evaluates the confidence that the said 
region is to be analyzed by itself. Following this evaluation, the agent communicates with 
the other agents in the litigious region and asks for their confidence value. The 
communicating agent can thereafter award the said region to the agent with maximum 
confidence. In one embodiment, the confidence value is computed by the agent's 
Processor. In another embodiment, the confidence value is attributed by a classifier such 
as a neural network using the agent's Neural Processor, where die neural network's 
activation value can be considered to weigh the confidence value. 
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5 Agent Adaptation. Collaboration and Reproduction 

To provide increased robustness and intelligence to the system, the devised agents within 
the present multi-agent system have been incorporated means to dynamically adapt 
themselves to any image they analyze. 

A first type of adaptation is the context based agent adaptation, where an agent adapts its 

10 parameters and goals based on the information the agent gathers at a higher semantic or 
spatial scale or based on the information provided by another agent found at a higher 
semantic or spatial scale. The agent first begins analyzing the image at its current scale 
level, then, to validate that the segmentation is precise and correct the agent gathers 
information from a higher scale which provides detail on the location and nature of the 

15 object being segmented. By having knowledge on the nature of the object, as provided 
from a higher scale analysis, the agent's behavior and actions can be dynamically 
modulated and adapted to improve the current segmentation, if required. As stated above, 
the gathering of information at a higher level can be achieved by the considered agent or 
by another agent at this higher level. In the embodiment of gathering higher level 

20 information, the process involves the steps of: 1) Determining the scale level at which 
additional information should be gathered, 2) Proper agent initialization, 3) Information 
gathering, and 4) Agent adaptation. In a preferred embodiment, the adequate scale level is 
determined before-hand. The second step of agent initialization can take many forms, 
such as, without limitation, the agent moving itself to the higher scale, the agent cloning 

25 itself to the higher scale, and instantiating new agents (can be of different specialization) 
at the higher scale by reproduction or other means. In the first case where the agent at 
scale level i (74) moves to the scale level n (76), the later analyzes the region of interest 
at the higher scale to gain new knowledge on the feature or object being analyzed. 
Referring to Figure lO.b, this process requires the agent to first identify the coordinate 

30 which it is currently at within the image (78), then load the higher scale image 
representation from memory (80), set it as the current image of interest (82), begin 
analyzing the image at the previously identified coordinate (84), and finally update its 
memory and adapt its parameters (86). In the other case, the agent clones itself at the 
higher level and continues the analysis through direct collaboration with its higher level 



14 



5 clone(s). This is similar to the just above stated method, but differs with the aspect of 
collaboration. Instead of a sequential process where the agent gathers information then 
adapts itself, the cloning method provides a parallel and dynamic adaptation method 
where the lower level agent dynamically adapts itself from the feedback provided by its 
higher level clones. Referring to Figure lO.c, this method requires the agent to first 

10 instantiate a new agent of same specialization (88), copy its mental state and memory 
content to the new agent (90) so that the later becomes a clone (copy), position the clone 
at the proper coordinate at higher scale (92), have the clone analyze the image (94) and 
communicate the gathered information to the lower level agent (96). In a preferred 
embodiment, an agent can simply invoke its "clone" class method to clone itself, wherein 

15 the said class method is preprogrammed to generate a copy of an instance of an agent. 
The other method provided for proper agent initialization is the instantiation of new 
agents at the higher scale. As stated above, this can be achieved by simple instantiation or 
reproduction. In such case, new agents of same or different specialization will be 
instantiated and positioned at the proper coordinate on the higher scale image by the 

20 lower level agent These new agents will thereafter analyze the image region and provide 
new information to the lower level agent by means of communication. The overall 
process is identical to the previously defined method, wherein the only difference lies in 
the type of agents that are instantiated. In the context of agent instantiation by means of 
reproduction, it is possible to instantiate new agents of specific specialization depending 

25 on the reproduction process of a specific agent. In the present invention, agent 
reproduction can be general or specialized, where for instance in one embodiment an 
agent specialized to detect object contours has the possibility of reproducing itself in 
agents specialized in analyzing image background and object regions. By providing 
specialized reproduction schemes, agents have the possibility of collaborating and 

30 gaining new knowledge optimally and specifically for the tasks they are required to do. 

Another type of adaptation is based on reinforcement The agent begins a first analysis, 
and then after a given number of pixels have been analyzed, the agent reiterates the 
analysis from the start. This reiteration allows for the agent to iteratively increase its 
knowledge on the information that is being analyzed and also offers the possibility of 
35 exploring differing analysis methods. This method allows an agent to incrementally 
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5 update its knowledge and understanding of the object or feature it is currently analyzing 
and can therefore optimally adapt its analysis parameters in accordance to its current 
knowledge. The adaptation involved can be with respect to the parameters relating to the 
Processor component or the Sensors, where for instance the agent can modulate its noise 
sensitivity, adjust its threshold parameters, and vary the scale and resolution of its 

10 sensors. 

Detailed description of the Invention in the preferred embodiment of 
Satellite Image Analysis 

A detailed description of the present invention will be further described hereunder using 
the example of detecting entities such as industrial buildings and bridges in satellite 
IS images, as well as further providing a symbolic interpretation of the image landscape. 

Firstly, the system reads a digital satellite image by means of its input device, as depicted 
by numeral 2 in Figure 1. An illustration of a satellite image is depicted by Figure 1 1, 
which contains bridges, buildings, roads and water courses. Then, the Program (4) 
preprocesses the digital image with objective of extracting the predominant water 

20 structures within the image which will thereafter serve as the initialization point for the 
multi-agent system. A possible method for extracting the main structures is to simply 
smooth the image at a high level by convolution with a Gaussian kernel followed by a 
threshold of pixels so as to only keep the pixels that are part of main water structures. 
Then Ate ridges of these main structures are extracted by means of a ridge extraction 

25 operator. Figure 12.a depicts the extraction of the most predominant water courses along 
with their ridges. Referring to Figure 12.b, following these preprocessing operations, the 
Program instantiates three agents specialized for the following of ridges, in accordance to 
the three ridge entrance points (98,100,102). The Program extracts the coordinates of 
these entrance points and assigns the agents respectively at these coordinates within the 

30 image. The objective of the ridge agents is to follow the ridge that serves as a path and 
thereafter collaborate with other agents if required. Once these agents have been 

initialized and assigned, the multi-agent system becomes autonomous. In the next step, 

* 

the ridge agents initialize border agents that will analyze the border of the water 
structures. The border agents are not directly guided by a path but instead use their 
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5 sensors to orient their selves in the proper direction. The border agents are specialized to 
respond to bimodal input from their sensors as they are expected to perceive pixel values 
within a specific range on one side and pixel values with a different intensity range on the 
other side* This is from the fact that pixels that are part of the water structures will have a 
different intensity then the land or other non-water structures in the image. Based on this 

10 a priori knowledge, the border agent will be able to recognize water borders and move 
within the image in accordance, while at the same time segmenting the borders for 
thereafter allowing further analysis. 

The next step consists in the agents to move within the image and analyze its content. 
Referring to Figure 13, during the ridge path following, the ridge agents will encounter a 

15 forking in the path (104,106,108). When a ridge agent identifies the problem, it generates 
offspring of same type in a number according to the number of branching paths. These 
offspring will continue to follow ridges in an autonomous manner. At the same time, 
border agents analyze the image using their sensors and move along the water course 
borders. Referring to figures 14.a and 14.b, the border agents will encounter a bridge 

20 (1 12) in their path which will be seen by the agent as a path discontinuity since the bridge 
line traverses the water and part of the bordering land (1 14). This discontinuity will be 
identified by the concerned agent (1 10) and will trigger that adaptation of the agent's 
sensors to verify the nature of this discontinuity. The agent's sensors will successively 
increase in size and scale representation until the border edge on the other side of the yet 

25 unknown discontinuity (116) is sensed by the agent At this point, the agent reproduces 
itself in a new Line Agent that is initialized on the line discontinuity. The objective is to 
allow this new agent to gather more information on the latter. At this point, the Border 
agent has identified a discontinuity that crosses the border which provides an insight on 
the nature of the latter can possibly be a building, a port, a road or a bridge. The new line 

30 agent will provide the border agent with enough information to make the proper 
identification of the structure. Now that the line agent is initialized it begins the analysis 
and following of the line structure. This specific agent performs basic edge enhancement 
to compute the necessary information to move in the proper path direction and 
simultaneously analyses the pixel values on both sides of the line (118). By detecting and 

35 ' analyzing these pixels, the agent can verify whether the pixels correspond to water or 
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5 land. Since in the present case the pixels along the path correspond to water, the line 
agent communicates with the Border agent and informs it that the discontinuity structure 
corresponds to the entity bridge" The line agent continues its path until it reaches the 
end of the line, where if it encounters another border agent (120), it can also inform it on 
the nature of the current line structure. 

10 The next process in the analysis consists in providing a symbolic and semantic 
interpretation of the water courses. In such case, the agents are required to collaborate at 
various spatial scales in order to properly detect and understand the present structures, 
without being disrupted by noise, artifacts, or other dominant structures. Relating to 
Figure 15, we can notice the presence of primary (122) and secondary (124) water 

15 courses that are to be detected, segmented and interpreted by the system. First, in 
continuation with the previous processes, we see that the ridge agents come to a point 
where their paths meet (126-128, 130-132). This contradiction triggers negotiation 
amongst the considered agents. In this case, as the agents correctly follow the same path, 
the negotiation will result in a fusion of both considered agents (126 and 128, 130 and 

20 132). Referring to Figure 15.b, the resulting fused agents (134, 136) will contain the 
knowledge acquired throughout the analysis of both agents, which means that the agent 
will have in memory the path of the complete water course ridge. This fusion results in 
the generation of the symbolic concept of "same" meaning that this path consists of a 
single water course. Now that there remains two fused ridge agents, another level of 

25 interpretation can be provided. Both agents (134 and 136) have the knowledge that there 
initially was a split in path and that the result was a fusion of agents since they were on a 
same path. From this information, the agents* State Machines will provide the conclusion 
that both agents may be part of a same water course, and that the separation may be 
caused by an entity such as an Isle, platform, or ship, for instance. To validate this, both 

30 agents instantiate an agent specialized in detecting blob structures at high spatial scales. 
Referring to Figure 16a, this is achieved through collaboration by first selecting a high 
spatial scale representation of the current image and instantiating both new agents (138, 
140) at the coordinates of their respective "parents" in the new image. Their goal is to 
detect a blob structure (142) in the vicinity of their initial coordinates. The two lower- 

35 scale parent agents restrict the possible displacements of the higher-scale agents by 
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confining them within the range of both agents (144), based on their knowledge that the 
structure is between them. Once initialized, the agents go about analyzing the region to 
detect and segment the "blob" structure. Further in the process, both blob agents will try 
to segment the same pixels, which indicates that they are detecting the same structure. 
From this, the lower-scale agents will be provided with the knowledge that this is in fact 
the same water course containing an entity. At this step, the system may chose to 
continue the blob detection in order to provide symbolic and semantic interpretation of 
this entity. In the present case, based on the shape and spectral intensity of the entity, the 
agents can conclude that it consists in an Isle or sand bank. In another scenario, this entity 
could be a ship, where in such case, the blob agent would detect that it is a ship based on 
the entity's specific shape by means of its state machine and memory unit To achieve 
this identification, the agents are first required to precisely segment the structure and 
extract shape descriptors, such as maximum and minimum diameters, as well as spectral 
descriptors. By inputting this information in both or either their Neural Processor and 
State Machine, the agents will have the, possibility of precisely identifying the analyzed 
structure and assign it a symbolic value, as mentioned previously. 

Throughout these, processes the border agents go upon segmenting the water borders. 
Now that the ridge agents have identified a single water course, the border agents 
collaborate with the later to segment the whole water course. The objective is to relate 
both borders to the appropriate water course, to quantify and analyze the water course 
and to thereafter identify connecting courses. Quantifying and analyzing the water course 
requires the agents to simultaneously perform the analysis at multiple spatial scales so as 
to gather the most precise and robust information, allowing for further detecting 
structures within and on the periphery of the water course. First, one of the remaining 
ridge agents positions itself at the beginning of the ridge path and displaces itself at the 
same coordinate in a higher spatial scale representation. Its objective is to inform the 
agents to which ridge and water course they belong to and to instantiate Water Agents 
that will analyze the water course. From its higher scale perception of the structures, the 
ridge agent readily sees that the border agents directly belong to the same water course. 
As the ridge agent moves along the ridge path it instantiates water agents. These new 
agents are specialized to analyze water structures by means of their multiscale sensors. 
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5 These agents' sensors simultaneously perceive the locally sensed pixels at multiple 
spatial scales. This multiscale view generates a multiscale "signature" that is directly 
analyzed by their Processor and Neural Processor components. The Neural Processor will 
classify the signatures according to these two classes: water and non-water. This simple 
binary classification allows the agent to take the appropriate actions upon the 

10 classification output If the Neural Processor identifies a signature as non-water, then the 
agent may be in the presence of an entity such as a ship. In a specific embodiment, the 
water agent can reproduce itself in agents specialized to segment such entities. The water 
agents are constrained by the borders previously segmented by the bonier agents, which 
prevents them from segmenting and analyzing structures that may have similar signatures 

15 as water pixels. Furthermore, since the water agents sense the image at multiple scales, 
they are robust to noise present within the image. At the end of the process, the water 
agents will have segmented the complete water course and other entities will have been 
further segmented and analyzed by other agents. The collaboration of border agents and 
water agents also further allows for the segmentation and identification of bordering 

20 structures such as buildings. In this case, the buildings will be seen as small 
discontinuities by the border agents, while at the same time perceived as non-water 
entities by the water agents. The water agents can therefore instantiate other specialized 
agents to segment and analyze these bordering entities. At the same time, the border 
agents can constrain the analysis domain of these new agents. 

25 Overall, by letting evolve the previously described processes, the entire image and image 
structures will be precisely segmented, quantified and characterized. The agents will go 
upon achieving their goals and the overall autonomous collaboration will result in the 
complete quantification and understanding of the therein contained structures. The final 
step of the overall process consists in storing the segmented image as well as the 

30 symbolic interpretation of every identified structure in a repository such as a database. 
The assignment program can execute this task by accessing the information contained in 
memory and appropriately storing it in a database. This stored information can be used by 
other programs, such as data-mining programs and display program, and can be directly 
interpreted by users. 
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While the invention has been described in connection with specific embodiments thereof, 
it will be understood that it is capable of further modifications and this application is 
intended to cover any variations, uses, or adaptations of the invention following, in 
general, the principles of the invention and including such departures from the present 
disclosure as come within known or customary practice within the art to which the 
invention pertains and as may be applied to the essential features hereinbefore set forth, 
and as follows in the scope of the appended claims. 
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5 WHAT IS CLAIMED IS: 

1. A method for analyzing one or a plurality of digital images using a plurality of 
software agents, comprising: 

- A program for initializing and assigning a plurality of agents to at least one 
of said images; 

10 - Means for said agents to perceive at least one of said images at one or a 

plurality of scale representations; and 

- Means for said agents to modify pixel values of said images. 

2. The method as claimed in claim 1 , wherein each said agents further comprise: 

15 - Sensors to locally sense information comprised in said images; 

- Decision making means for allowing said agents to take action in 
accordance to specific mental states; 

- Processing means for processing said information perceived by said sensors. 

20 3. The method as claimed in claim 1, wherein said step of assigning further 
comprises the steps of: 

- Preprocessing at least one of said images; 

- Producing spatial coordinates from results of said step of preprocessing; 

- Assigning said agents according to said spatial coordinates. 

25 . 

4. The method as claimed in claim 2, wherein said agents have the capability to 

further adapt said sensors by changing configuration of said sensors. 

5. The method as claimed in claim 4, wherein said step of changing configuration of 
said sensors comprises at least changing size of said sensors. 

♦ 

30 6. The method as claimed in anyone of claims 4 and 5, wherein said step of changing 
configuration of said sensors comprises changing resolution of said sensors. 

7. The method as claimed in claim 1 , wherein said means to perceive comprises: 

- Analyzing said images at a first scale representation by means of at least one 
of said agents; 

35 - Analyzing said images at a second scale representation by means of at least 

one other of said agents; and 

- Exchanging information between said agents at said first and second scale 
representations. 
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8. The method as claimed in claim 7, wherein said step of exchanging information 
further allows for at least one of said agents to adapt itself. 

9. The method as claimed in claim 1 , wherein said scale is a spatial scale. 

10. The method as claimed in claim 1, wherein said scale is a semantic scale. 

11. The method as claimed in any one of claims 2, 4, 5,6, wherein said sensors 
provide a spatial multiscale representation of said image, wherein said multiscale 
representation simultaneously provides said agents with image infonnation at 
every considered spatial scale. 

12. The method as claimed in claim 11, wherein said multiscale information provides 
a multiscale signature and where ( in said signature further allows said agent to 
classify said information. 

13. The method as claimed in claim 12, wherein said step of classifying information is 
achieved by means artificial neural networks. 

14. A method for contour recognition in one or a plurality of digital images by means 
of one or a plurality of software agents, comprising the steps of: 

- Initializing and assigning at least one of said agents to an image, said agents 
comprising at least one sensor; 

- Said agents recognizing said contours in said image by means of said 

sensors; and 

- Said agents drawing said recognized contours in said image. 

15. An apparatus for analyzing one or a plurality of digital images, comprising: 

- Image data input means; . . , 

- At least one processing node, said processing node comprising memory and 

processing unit; . . „. 

- A method for analyzing said images, said method as claimed in claim 1, and 

- Display means for displaying said images. 
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Abstract 



A method for analyzing one or a plurality of digital images using a plurality of software 
agents, comprising: a program for initializing and assigning a plurality of agents to at 
least one of said images; means for said agents to perceive at least one of said images 
at one or a plurality of scale representations; and means for said agents to modify pixel 
values of said images. 
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Figure 1: Agent initialization and assignment 



Figure 2: Agent Components 
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Figure 3: Example Agent assignment 
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Figure 5: Agent Sensors in portion of image 
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Figure 6: Agent communication 
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Figure 9 a: Multiresolution Sensing Flow chart 





Figure 9 b: Multiresolution Sensing Illustration 
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Figure 10 a: Higher scale information gathering by single agent 
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Figure 16b: Higher scale blob detection 
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Figure 17: Apparatus with multiple processing nodes 
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